Basic Data Wrangling and Data Visualization in R

class: center, middle, title-slide

# Basic Data Wrangling and Data Visualization in R
### Luke Tierney
### University of Iowa
### 21 June, 2021

---

class: center, middle

# Introduction

---
## Outline

In this class I will

* Briefly outline the history of R.

* Using some examples briefly show how to do data wrangling
  and visualize data in R.

Materials for this class are available on GitHub at
<https://github.com/ltierney/SIBS-WV-2021.git>.

* You can access it as an RStudio project by following the menu selection
  **File > New Project > Version Control > Git** and specifying this URL.

* You can use the `git` command line client with
    ```shell
git clone https://github.com/ltierney/SIBS-WV-2021.git
    ```

Materials for our _Data Visualization and Data
Technologies_ course are available at

<http://www.stat.uiowa.edu/~luke/classes/STAT4580-2021/>

---

## Tools

Some tools I will be using:

* The [RStudio](https://www.rstudion.com) IDE.

* Many features from the basic [R](https://www.r-project.org) distribution.

* Some tools from the [_tidyverse_](https://www.tidyverse.org/).

* The [`ggplot`](https://ggplot2.tidyverse.org/) package based on
  the _Grammar of Graphics_ framework.

Most of the packages are loaded by loading the `tidyverse` package:

```r
library(tidyverse)
```

---
## References

Useful references:

> Hadley Wickham and Garrett Grolemund (2016), [_R for Data
> Science_](http://r4ds.had.co.nz/), O'Reilly.

> Claus O. Wilke (2019), [_Fundamentals of Data
>  Visualization_](https://serialmentor.com/dataviz/), O'Reilly.

> Kieran Healy (2018) [_Data Visualization: A practical
> introduction_](http://socviz.co/), Princeton

> Rafael A. Irizarry (2019), [Introduction to Data Science: _Data
> Analysis and Prediction Algorithms with
> R_](https://rafalab.github.io/dsbook/), Chapman & Hall/CRC. ([Book
> source on GitHub](https://github.com/rafalab/dsbook))

There is also a small interactive [tutorial](../tutorial/penguins.Rmd) available.

**Ask questions any time!**

---
class: center, middle

# The R Language

---
## Background

R is a language for data analysis and graphics.

* R was originally developed by Robert Gentleman and Ross Ihaka in the
  early 1990's for a Macintosh computer lab at U. of Auckland, New Zealand.

* R is based on the S language developed by John Chambers and
  others at Bell Labs.

R is an Open Source project.

* Since 1997 R is developed and maintained by the R-core group,
  with around 20 members located in maor than 10 different countries.

* R is widely used in the field of statistics and beyond, especially in
  university environments.

* R has become the primary framework for developing and making available
  new statistical methodology.

* Many (now over 17,000) extension packages are available through CRAN or
  similar repositories.

---
## Working with R

R is designed for interactive data exploration.

* Interaction is through a _read-eval-print loop (REPL)_.

* This is also called a _command line interface (CLI)_.

All computations are specified in the R language.

* Even for simple tasks you need to know a little of the language.

* After learning to do simple tasks you know some of the language.

The language is used to

* prepare data for analysis;

* specify individual analyses;

* program repeated or similar analyses;

* program new methods of analysis.

Specifying these tasks in a language supports _reproducible research_.

---
## Working with R

The R language operates on vectors and arrays.

Commonly used data types are:

* integer and numeric vectors;

* logical vectors;

* character vectors;

* factors.

All basic vector types support missing (`NA`) values.

Arithmetic operations are vectorized to operate element-wise on vectors.

Data vectors are usually combined into table-like objects called _data
frames_.

---
## The Data Analysis Process

A figure that shows the steps usually involved in a data analysis
project:

<center>
<div id="htmlwidget-586dc7e500d17004a9c4" style="width:576px;height:360px;" class="nomnoml html-widget"></div>
<script type="application/json" data-for="htmlwidget-586dc7e500d17004a9c4">{"x":{"code":"\n#fill: #FEFEFF\n#lineWidth: 1\n#zoom: 4\n#direction: right\n\n#padding: 25\n#fontsize: 18\n#fill: #E1DAFF; #D4A9FF\n#stroke: #8515C7\n#linewidth: 2\n\n[Import] -> [Understand]\n[Understand |\n [Wrangle] -> [Visualize]\n [Visualize] -> [Model]\n [Model] -> [Wrangle]\n]\n[Understand] -> [Communicate]","svg":false},"evals":[],"jsHooks":[]}</script>
</center>

These steps are often repeated many times, so it is important to make
your work reproducible.

---
## Reproducible Data Analysis

Making your work reproducible:

* Save you work in a text file or notebook.

* Track changes to your files with a version control system like
  [`git`](https://git-scm.com/).

* Use a system like [Rmarkdown](https://rmarkdown.rstudio.com) to
  prepare your reports.

This allows you to re-create your report when data changes (as it
often will!).

A good resource for setting up your tools to support this is [_Happy
Git and GitHub for the useR_](https://happygitwithr.com/).

---
class: center, middle

# Some Examples

---
## Example Data Sets

When working with research data, a first step is usually to read and
clean the data.

We'll put that off for a little while and work with some data sets
made available in R packages.

Data sets available in R packages include:

* many classic data sets;

* newer, often larger, data sets useful for learning;

* current data obtained by querying web APIs.

---
## Old Faithful Eruptions

A simple classic data set is the `geyser` data frame available in
package `MASS`.

.pull-left[

```r
data(geyser, package = "MASS")
dim(geyser)
## [1] 299 2
head(geyser, 4)
## waiting duration
## 1 80 4.016667
## 2 71 2.150000
## 3 57 4.000000
## 4 80 4.000000
```
]
--
.pull-right[
<div class = "note">
`head` and `tail` return the first and last few rows of a data frame.

They are useful for quick sanity checks.
</div>

]

The rows represent measurements recorded for eruptions of the [_Old
Faithful_](https://www.yellowstonepark.com/things-to-do/geysers-hot-springs/about-old-faithful/)
geyser in Yellowstone National Park, Wyoming.

The variables are:

* `waiting`: the time in minutes since the previous eruption;

* `duration`: the duration in minutes of the eruption.

---
## Old Faithful Eruptions

The durations have a bimodal distribution:

.pull-left[
![](slides_files/figure-html/geyser-hist-1.png)
]
--
.pull-right[

```r
ggplot(geyser) +
    geom_histogram(aes(x = duration),
                   bins = 15,
                   color = "black",
                   fill = "grey")
```

{{content}}
]
--
This is [`ggplot`](https://ggplot2.tidyverse.org/) code for creating a
histogram.

{{content}}
--

<div class = "note">
A basic template for creating a plot with `ggplot`:

```r
ggplot(data = <DATA>) +
 <GEOM>(mapping = aes(<MAPPINGS>))
```
</div>

---
## Old Faithful Eruptions

An interesting question is whether the duration can be used to predict
when the _next_ eruption will occur.

A plot of the _previous_ duration against the waiting time to the
current eruption:

.pull-left[
![](slides_files/figure-html/geyser-scatter-1.png)
]
--
.pull-right[

```r
ggplot(geyser) +
    geom_point(aes(x = lag(duration),
                   y = waiting))
```

{{content}}
]
--
It looks like a useful rule would be to expect a shorter waiting time
after a shorter eruption duration.

---
## Old Faithful Eruptions

An interesting feature:

Many durations are recorded as 2 or 4 minutes.

This can also be seen in a histogram with a small bin width:

.pull-left[
![](slides_files/figure-html/geyser-hist-narrow-1.png)
]
--
.pull-righ[

```r
p <- ggplot(geyser) +
 geom_histogram(aes(x = duration,
 y = stat(density)),
 fill = "grey",
 color = "black",
* binwidth = 0.1)
p
```

{{content}}
]
--

<div class = "note">
`ggplot` produces a plot object.

Drawing only happens when the object is printed.
</div>

---
## Old Faithful Eruptions

Does this rounding matter?

* For many analyses it probably doesn't.

* It might if you wanted to fit normal distributions to the two groups.

.pull-left[
Taking 3 minutes as the divide between short and long durations we can
first pick out the short and long durations:

```r
d <- geyser$duration
d_short <- d[d < 3]
d_long <- d[d >= 3]
```
]
--
.pull-right[
Then compute the means and standard deviations as

```r
mean(d_short)
## [1] 1.980317
sd(d_short)
## [1] 0.2779829
mean(d_long)
## [1] 4.262113
sd(d_long)
## [1] 0.3937525
mean(d >= 3)
## [1] 0.6488294
```
]

---
## Old Faithful Eruptions

An approach that scales better:

Compute group summaries using tools from the `dplyr` tidyverse
package.

First, add a `type` variable:

```r
geyser <- mutate(geyser, type = ifelse(duration < 3, "short", "long"))
```

The summaries can then be computed as

.pull-left[

```r
sgd <- summarize(group_by(geyser, type),
 mean = mean(duration),
 sd = sd(duration),
 n = n())
(sgd <- mutate(sgd, prop = n / sum(n)))
## # A tibble: 2 x 5
## type mean sd n prop
## <chr> <dbl> <dbl> <int> <dbl>
## 1 long 4.26 0.394 194 0.649
## 2 short 1.98 0.278 105 0.351
```
]
--
.pull-right[
<div class = "note">
The functions `summarize`, `group_by`, and `mutate` are from the
`dplyr` package that implements a _grammar of data manipulation_.
</div>

]

---
## Old Faithful Eruptions

This computation can also be written using the _forward pipe operator_ `%>%`:

```r
sgd <-
 group_by(geyser, type) %>%
 summarize(mean = mean(duration),
 sd = sd(duration),
 n = n()) %>%
 ungroup() %>%
 mutate(prop = n / sum(n))
sgd
## # A tibble: 2 x 5
## type mean sd n prop
## <chr> <dbl> <dbl> <int> <dbl>
## 1 long 4.26 0.394 194 0.649
## 2 short 1.98 0.278 105 0.351
```
--

<div class = "note">
The pipe operator allows a sequence of operations to be chained together.

The left-hand operation is passed implicitly as the first argument to
the function called on the right.
</div>

---
## Old Faithful Eruptions

One way to show the superimposed normal densities:

.pull-left[
![](slides_files/figure-html/geyser-hist-dens-1.png)
]
--
.pull-right[

```r
f1 <- function(x)
 sgd$prop[1] * dnorm(x, sgd$mean[1], sgd$sd[1])
f2 <- function(x)
 sgd$prop[2] * dnorm(x, sgd$mean[2], sgd$sd[2])
p <- p +
 stat_function(color = "red", fun = f1) +
 stat_function(color = "blue", fun = f2)
p
```

{{content}}
]
--

<div class = "note">
A `ggplot` can consist of several _layers_.
</div>

---
## Old Faithful Eruptions

The means and standard deviations are affected by the rounding.

Summaries that omit values equal to 2 or 4 minutes can be computed as

```r
geyser2 <- filter(geyser, duration != 2, duration != 4)
sgd2 <-
 group_by(geyser2, type) %>%
 summarize(mean = mean(duration),
 sd = sd(duration),
 n = n()) %>%
 ungroup() %>%
 mutate(prop = n / sum(n))
sgd2
## # A tibble: 2 x 5
## type mean sd n prop
## <chr> <dbl> <dbl> <int> <dbl>
## 1 long 4.36 0.422 141 0.632
## 2 short 1.97 0.315 82 0.368
```

---
## Old Faithful Eruptions

A plot showing curves computed both ways:

--
.pull-left[
![](slides_files/figure-html/geyser-hist-dens-2-1.png)
]
--
.pull-right[

```r
f1_2 <- function(x)
 sgd2$prop[1] * dnorm(x, sgd2$mean[1], sgd2$sd[1])
f2_2 <- function(x)
 sgd2$prop[2] * dnorm(x, sgd2$mean[2], sgd2$sd[2])
p <- p +
 stat_function(color = "red",
 linetype = 2,
 fun = f1_2) +
 stat_function(color = "blue",
 linetype = 2,
 fun = f2_2)
p
```
]